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ABSTRACT 



This paper discusses issues surrounding the use of licensure 
examination results as measures of outcome in institutions of higher 
education. Based on informal telephone surveys of several state university 
systems, and on information obtained from national certification 
organizations, faculty experts, reviews of the literature, and the experience 
of the State of Florida, the study finds no uniformity of opinion regarding 
the use of certification and licensure examination results to assess academic 
programs; examinations are seen more as a measure of student ability than 
program validity. Appendixes include: a list of national exomin=i^”ion agencies 
by discipline; brief synopses of survey results of state universities and 
systems conducted from December 1997 through March 1997; a table of national 
licensure information by discipline; and a short monograph on the State of 
Florida's experiences in obtaining pass rate data. (CH) 
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Licensure Examination Results as Outcomes Indicators: Issues and 



Challenges 



Abstract 

As accountability and performance based budgeting become 
increasingly prevalent in higher education, quantifiable outcomes 
measures have gained in popularity. A measure that has initial 
appeal is licensure examination results. However, the measure is 
not yet widely used, due primarily to difficulties in obtaining 
data, and is fraught with issues which need to be addressed in 
order to obtain meaningful information. The paper discusses the 
extent to which the measure is used in several systems of higher 
education, the issues surrounding pass rates on examinations, and 
provides a list of national organizations which administer 
licensure examinations, to assist researchers interested in the 
use of this measure. 

Introduction 

Outcomes measures which may be obtained with relative ease 
and not much additional expenditure have gained popularity as 
increased pressure has been applied to colleges and universities 



to provide quantifiable data about their programs and graduates. 
The results of licensure examinations as an indicator of outcomes 
has a particular appeal because it is a quantitative measure and 
is directly related to the employability of an institution' s 
graduates. While licensure examination results can prove useful 
as an outcomes measure and as a means to curricular improvements 
in academic programs, the ease with which the data may be 
obtained varies widely by state and by discipline. In addition, 
a number of issues should be taken into consideration when 
interpreting the data in order to understand the meaning of the 
information gathered. 

The paper presents and discusses issues which should be 
taken into consideration and the challenges faced in the use of 
licensure examination results as a measure of outcomes in 
institutions of higher education. The discussion is based on 
telephone surveys of several state university systems, national 
entities responsible for certification examinations, faculty 
experts in various disciplines, review of publications and 
documents regarding licensure, and experiences in Florida, a 
state which has had greater access to licensure information than 
many others . 
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Certification and Licensure Exeuninations 



What are certification and licensure examinations, and what 
is the difference between the two? Both types of examinations 
test the knowledge of an individual in a particular discipline, 
to assure a certain level of competence. Licensure is granted by 
the state, whereas certification is granted by a professional 
organization, and a certification examination may be developed 
and administered by a national entity such as a commission, a 
council or an association. Some national associations contract 
with a testing service, such as the Educational Testing Service 
(ETS) to develop and administer the examination. Passing a 
national certification examination is often one criteria for 
obtaining state licensure. In the case of some disciplines, a 
state may use a national examination and supplement it with a 
state-specific portion of the examination, usually dealing with 
the state laws relating to the profession. Such examinations may 
be called licensure examinations. Some professions may have 
certification examinations, but passage of the examination may 
not be required for employment and licensure may not exist (such 
as vocational rehabilitation counseling and health information 
management in Florida.) In this case, practitioners seek 
certification not because of necessity, but to reassure clients 
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of their capabilities. 



It is important to understand the purpose of licensure and 
certification examinations before using pass rates to make 
assumptions about the quality of educational programs. In most 
professions, certification or licensure is intended to ensure a 
threshold of competence on the part of the examinee to safeguard 
the public. While a pass rate that falls significantly below the 
national average should be cause for concern and further 
analysis, a high pass rate does not necessarily signal a high 
quality program. The educational programs should be preparing 
students to do far more than simply meet a threshold of 
competence. It is possible that two programs of similarly high 
pass rates have vastly different levels of quality, where one 
program simply prepares the students to pass the examination, 
while the other goes much further. 

Disciplines with National Exeuninations 

Disciplines offered at the bachelor's level or higher, in 
which national examinations are available and widely used include 
accounting, architecture, landscape architecture, interior 
design, clinical laboratory sciences (medical technology) , 
clinical social work, dentistry, dietetics and nutrition, 
engineering, law, marriage and family therapy, medicine, mental 
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health counseling, nursing, nursing home administration, 
occupational therapy, pharmacy, physical therapy, psychology, 
school psychology, radiological sciences, rehabilitation 
counseling, respiratory therapy, speech- language pathology and 
audiology, teacher education and veterinary medicine. Other 
disciplines which are offered at an associate degree level also 
have national certification examinations. In some instances, 
disciplines may have national examinations corresponding to 
programs at the associate and at the bachelor's level. In a few 
disciplines, such as nursing, both associate and baccalaureate 
degree students sit for the same examination. Appendix A 
contains a list of the national entities responsible for widely 
used certification/licensure examinations. 

Use of Examination Data by States 

An informal telephone survey of several state university 
systems revealed that some states are already using licensure 
pass rates as a performance indicator at the state level, either 
as part of an accountability report or performance based 
budgeting. These states include North Carolina, Tennessee, Texas 
and Wisconsin. A complete list of states surveyed is included in 
Appendix B. North Carolina uses pass rates as a component of the 
University Assessment Plan. They are also considering using it 
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as a measure for the Incentive Funding Plan. Tennessee uses 
pass rates as a measure for performance based funding. It is one 
of 10 variables, accounting for 5% of the budget. Tennessee has 
devised a formula for this purpose which uses pass rates and a 
sliding scale based on the number of first time test takers (pass 
rates with many takers are weighed more heavily) . The formula 
generates points for performance based funding. Pass rates are a 
measure in the performance based funding in Texas, although no 
funds are tied directly to this measure. Texas is planning to 
incorporate pass rates into the program review process . In 
Wisconsin, pass rates are one indicator on the accountability 
report. All these states only use pass rates on 
certification/licensure examinations in selected disciplines, 
such as nursing, law and medicine. The decision on which 
disciplines to use is based on ease of obtaining the data and how 
critical passage of the examination is to practice. 

In Florida, South Carolina, and Colorado, pass rates are one 
indicator being considered for performance based funding in the 
near future. Florida used pass rates as an indicator in the 
accountability report in 1993, but they have not been used since 
then. However, pass rates continue to be used as one indicator 



of program quality in systemwide program reviews (a practice of 



long-standing) . A few states such as California, Arizona and 
Georgia have rejected the concept of using licensure pass rates 
as an indicator. 

Problems encountered by state systems include: the limited 

availability of pass rates from national entities; the financial 
burden of obtaining pass rates from some agencies; the 
interpretation of the rates once obtained; and the non-existence 
of some type of national database or software coursefile where 
data can be entered, configured, and used for specific research 
interests . 

Data Considerations 

Accessibility. National and state agencies responsible for 
the examinations have traditionally viewed the examinations as 
tools for assessing the competence of individual test-takers, not 
as indicators of performance for the programs and institutions 
which prepared the test-takers. In fact, several agencies 
contacted for information cautioned against the use of pass rates 
for evaluating programs. Because of this viewpoint, obtaining 
data for program evaluation can be quite a challenge; many 
agencies are not "geared up" to provide such data. The extent to 
which useful data is collected, recorded, and made readily 



available varies greatly by discipline and agency. Some national 



associations routinely provide the data to the institutions. 



Others either do not collect data in a form that would be useful 
to the programs or do not make a practice of providing the data 
to the programs. Some charge a fee to provide the data. 

State systems or university central administrations may 
either request the affected programs to provide the pass rates to 
them, or obtain them directly from the national entity or state 
agency responsible for the examination. The benefit of 
requesting data from the programs is that this is probably the 
easier method since programs have better access to the 
information than a central administration or system office. The 
benefit of a central office obtaining data directly is that one 
has more control of the data included in the report. 

In Florida, obtaining data from the appropriate state 
agencies is often a challenge. The ideal situation would be for 
the appropriate agencies to collect the relevant information, 
store it in an easily accessible form on a database, whereby 
institutions of higher education could tap in to the database to 
not only obtain pass rates but also conduct more in depth studies 
which could examine the pass rates by cohort, by race and gender, 
subtests, the average number of attempts to pass all subtests, 
and other variables. In Florida, the State University System, 
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the Division of Community Colleges, and some private universities 



are attempting to work with the Department of Business and 
Professional Regulation (DBPR) to bring about access to the data 
through a database . 

Subtests. Many examinations, such as those in psychology, 
clinical social work, medical technology, speech pathology and 
accounting, consist of two or more subtests. Often in such cases 
a single overall pass rate for all subtests attempted will be 
reported. This means that a person who sits for two subtests and 
passes them both will be counted as a pass, and a person who sits 
for three subtests and passes two will be counted as a fail. 
However, in some disciplines such as architecture, only the pass 
rates for each subtest are reported. Examinees may retake only 
the subtests they failed. These factors make it difficult to 
interpret pass rates for repeat takers. Also, reporting pass 
rates on a number of subtests per discipline does not lend itself 
to brevity and simplicity which are important in reporting pass 
rates to the general public and legislators. For public 
reporting, it may be best to provide only the overall pass rate 
of first time takers (if available), rather than results on 
subtests and pass rates for repeat takers. Information on 
individual subtests is useful for curricular review purposes . 
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Test- takers. Individuals who sit for an examination for the 



first time are referred to as "first time takers." Those who 
retake the examination or subtests that they failed on the first 
try are "repeaters." Data is most useful if it distinguishes 
between first time takers and repeaters. Some national entities 
and state agencies provide data on these two types of test -takers 
separately, but some will only report first time takers, and 
others will combine the two. Combining first time takers and 
repeat takers is not advisable because it skews the data and 
makes useful interpretation difficult. Usually the repeaters 
tend to have significantly lower pass rates than the first time 
takers. Also, the relationship between the quality of the 
education received in the program and performance on the test is 
not as clear since time and other experiences have intervened. 

The most useful, most widely reported, and simplest measure to 
report is the pass rate of first time takers. That is, what 
percentage of the graduates from a particular program, taking the 
examination for the first time, received a passing score on the 
examination? In addition to this measure, it may be useful to 
know what proportion of your graduates do succeed in passing the 
examination within a reasonable period after graduation (e.g. 5 
years) . In ascertaining the economic impact of an institution's 
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programs on the state or local community it may be useful to have 
data on all graduates who eventually become certified or licensed 
in a field. 

Annual data. Most examinations are administered more than 
once per year. When obtaining data it is advisable to request 
sittings for a complete year. Reporting agencies may not always 
specify if they are providing you with data on all the sittings 
for a given year; it is best to probe, and also to specify what 
constitutes the beginning and ending of the reporting year. 
Evaluating Pass Rates 

A program's pass rate for a given year is the percentage of 
graduates from that particular program who received a passing 
score on all subtests attempted within the given year. Once pass 
rates are obtained, what does one do with them? It is not very 
illuminating to simply report pass rates for various disciplines 
for a given institution, without any points of reference. For 
example, whether a 75% pass rate is good news or bad news will 
depend on the discipline, and whether it is better or worse than 
previous pass rates for a particular program. The two reference 
points that are generally used are 1) national and/ or state 
pass rates for the examination, and 2) previous history of pass 
rates for graduates of the particular program. The passing 
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scores of the national examinations for most disciplines are set 



by the national agency responsible for developing, administering 
and scoring the examination. Individual states usually accept 
the passing score suggested by the national entity. If there is 
a change in passing score, it is usually increased by 5 points, 
thereby setting a higher standard for state competency and 
passage of the examination. Additionally, some states have 
developed their own section of the examination which tests 
candidates on the laws and rules of that state which govern the 
particular discipline. Normally, the passing score for this 
section of the examination is similar, if not the same, as that 
set for the national examination. Therefore, once the practice 
in a given state with regard to passing scores in various 
examinations is determined, one can make use of the national and 
state pass rate for comparison purposes. In most cases, national 
pass rates for various examinations do provide a useful point of 
comparison. These pass rates can be obtained through each 
national entity responsible for the respective examinations. A 
list of many of these national entities, their addresses and 
telephone numbers appears in Appendix A. National pass rates 
vary considerably by discipline, ranging from high rates of 90% 
and above for several allied health disciplines, to low rates of 
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20% such as the Certified Public Accountant (CPA) examination. 



Criticality for Practice in the Profession 

How critical is passing the certification or licensure 
examination for getting ones first job in the profession? This 
is an important question which should be asked before deciding 
which disciplines to include in a measure using pass rates as an 
outcomes indicator. Criticality can have a considerable impact 
on the pass rates themselves, as well as how important a measure 
of program quality the pass rates are. If passing the 
examination is not critical to obtaining their first job in the 
profession, students are not highly motivated to take the 
examination immediately upon graduation, or to perform very well 
on it. This may affect the pass rates through no fault of the 
educational program. Criticality can vary somewhat by state, 
depending on the state laws governing practice in the various 
professions. The criticality and other academic/licensure 
information, of each discipline, is listed in Appendix C. 

In general, there seems to be an interesting correlation 
between the national pass rates and criticality for employment. 
Professions in which criticality is high generally have high 
national pass rates, and professions in which criticality is 
moderate to low generally have moderate to low national pass 
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rates. In most of the health disciplines, criticality is high. 

In these disciplines, unless you pass the examination you cannot 
practice at a professional level in the discipline, and the 
financial consequences to the individual are severe. The 
expectation of almost all (if not all) students entering such 
programs is that they will pass the examination upon graduation. 
In contrast, in disciplines such as architecture and law the 
criticality is moderate, and in disciplines such as engineering 
and accounting, the criticality is relatively low. In law, for 
example, it is possible to become employed in many professional 
jobs in the field of law, other than practice as an attorney, 
without passing the bar examination. In engineering most 
students do not need to take the Fundamentals of Engineering 
examination for entry level positions in Engineering. One 
registered engineer may have a large number of nonregistered 
engineers working under his/her supervision. The criticality of 
being a registered engineer is higher in some engineering 
disciplines, however, such as civil engineering. In accounting, 
employers do not expect graduates to have passed the CPA 
examination prior to their first job. In both engineering and 
accounting, not passing the examination may be a barrier to 
advancement in certain types of jobs. Therefore individuals may 
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take the examinations a few years after graduation. This brings 



up the issue discussed next, i.e. the time elapsed between 
graduation and taking the examination. 

Time elapsed 

Some disciplines, such as architecture, engineering (to 
become a registered engineer) , and dietetics require an 
internship or some practical experience in the field prior to 
sitting for the examination. This experience is incorporated 
into some degree programs, while in others the experience occurs 
after graduation (this varies by discipline as well as by program 
within a given discipline) . The duration of the experience 
varies by discipline. In programs where the experience is gained 
after graduation, and the duration of the experience is a year or 
more, the more tenuous are the conclusions one can draw about the 
program based on the examination results. The likelihood that a 
significant portion of the graduates may not be reported back as 
part of the institution's pass rates increases, particularly when 
graduates (perhaps the most able ones) obtain work experience in 
other states or other countries. Also, the examination may be 
practice-based, and the results may be as much a reflection of 
the intervening practical experiences of the students as of the 
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academic program from which they graduated. 



Conclusion 

Opinions vary as to the validity of using certification and 
licensure examination results to assess academic programs. The 
overwhelming response to the authors' query of testing agencies 
regarding the usage of examination results to improve curricula 
was one of caution. They felt that the examinations are more a 
measure of the student's ability than of program quality. 

Certain agencies expressed their concern over the use of 
licensure pass rates as a measure of program quality, stressing 
that their examination is designed to test the candidates' 
practicum experiences as well as their educational background. 
For example, the examination in architecture is designed to test 
the knowledge of the candidate after the two to three year 
practicum that is required of all examination candidates. Some 
state systems of higher education and many academic programs 
within the Florida State University System have questioned the 
use of examination pass rates to judge the academic quality of 
programs. The discussions surrounding this issue in regard to 
accountability and performance based budgeting are in the early 
stages; in addition to the problems of obtaining data, there is 
uneasiness on the part of entities responsible for the 
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examinations as well as the academic community regarding the 
interpretation of the data. Based on experiences in program 
review and accountability, examination of data, discussions with 
entities which administer examinations, faculty, and other state 
higher education systems, the authors offer the following 
thoughts and recommendations: 

1. Pass rates above the national or state rates are not 
necessarily indicators of high quality educational programs. 
Certification and licensure examinations are generally intended 
to ensure a threshold of competence for the protection of the 
public. On the other hand, pass rates significantly below the 
national or state rates should always be cause for self -scrutiny 
in a program, and act as a trigger to examine the program for 
deficiencies, provided the number of test takers is not so low as 
to make the pass rate meaningless. There may be reasons for the 
low pass rates other than a problem with the quality of the 
program, but one should examine the relevant information before 
drawing conclusions or dismissing the pass rates. 

2. It is almost always useful for programs to review the 
examination results and pass rates of their graduates. Some 
disciplines, in which passing the examination is critical for 
practice, are more accustomed to such self -scrutiny than others. 
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Even in disciplines where passing the examination is not critical 



for practice, but many students enter with the goal of becoming 
certified or licensed, programs can glean useful information by 
examining the results, and should do so. Results for any one 
year may be an anomaly; a trend over several years would be more 
useful. After careful analysis of examination results and other 
relevant factors, programs should be willing to make curricular 
changes when necessary. 

3. To this end, there should be a greater willingness and 
ability on the part of entities responsible for developing, 
administering and reporting examinations, to collect and provide 
relevant aggregate data to programs. Organizations such as the 
Association for Institutional Research (AIR) and legislators 
interested in greater accountability could be of great help in 
creating a national milieu where aggregate data on examination 
results are more readily available. It would also be of 
assistance to institutional researchers if there was a central 
data bank which contained the national pass rates in various 
disciplines against which to compare program pass rates. 

3. A distinction should be made between scrutinizing pass rates 
at the program level and reporting pass rates at a state level 
for accountability and performance based budgeting purposes. At 
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the program level, the many complexities and caveats peculiar to 
a particular discipline, such as low criticality, time elapsed 
due to field experience requirements, etc. can be taken into 
account when interpreting the pass rates. Examining pass rates 
within the context of program reviews which involve external 
experts in the discipline is quite useful and brings a level of 
objectivity to interpreting a program's pass rates. At a state 
level, where examination pass rates are one of several measures 
reported, and the audience is not given to mulling the 
complexities of individual disciplines, it is best to limit 
reporting to disciplines where the criticality of passing the 
examination is high and no significant time intervenes between 
graduation and the examination due to a required field 
experience. Here the reporting can be relatively 
straightforward, and should include the annual pass rate for 
first time takers and a reference point such as a national or 
state pass rate for first time takers. Attempting to include 
information that does not lend itself to simplicity and brevity 
in an arena ruled by sound bites is likely to foster 
misunderstanding . 

4. If pass rates reported for accountability and performance 
based budgeting are limited to disciplines in which the data is 




20 



relatively straightforward (high criticality, no significant time 
lapse after graduation) , is it reasonable to judge the entire 
institution, and perhaps allocate a portion of the institution's 
budget, based on the performance of students in such a limited 
number of programs? The majority of the programs are likely to 
be in one discipline (health) , which further limits the 
applicability. 

5. Rather than individuals removed from higher education making 
judgements about entire institutions based on simple pass rates 
for a few programs, it may be preferable to hold institutions 
accountable, and they in turn hold programs accountable, for 
carefully scrutinizing the pass rates in relevant disciplines, 
and making changes when they are warranted. 
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Appendix A 



National Examination Agencies 



|AC©OUNTING 


American Institute of Certified Public Accountants 
201 Plaza Three 
Harborside Financial Center 


■ :S, .• . f, . i 


Jersey City, NJ 0731 1 
(201) 938-3429 



ARCflTEGTUlitE , 


National Council of Architectural Registration Boards 
1735 New York Avenue, N.W., Suite 700 


# -'.r 4^ ■■ i 


Washington, DC 20006 
(202) 783-6500 



CLINIGAL SOGIAL 
WORK 


American Association of State Social Work Boards 
400 South Ridge Parkway, Suite B 
Culpeper, VA 22701 
(540) 829-6880 



DENTISTRY 


Joint Commission on National Board Dental 
Examinations 




Southern Regional Testing Agency, Inc. 
303 - 34th Street, Suite 7 




Virginia Beach, VA 23451 
(804) 428-1003 



DIETETICS 


Commission of Dietetic Registration 




216 West Jackson Boulevard 
Chicago, IL 60606-6995 
(312)899-0040 



ENGINEERING 


National Council of Engineering Examiners and 

Surveyors 

P.O. Box 1686 

Clemson, SC 29633-1686 




(803) 654-6824 



INTERIOR DESIGN 


National Council for Interior Design Qualification 




50 Main Street 

White Plains, NY 10606-1920 
(914)948-9100 



National Examination Agencies 



LANDSCAPE : 


Council of Landscape Architectural Registration Boards 


ARCliTECTUi^E 


12700 Fair Lakes Circle, Suite 1109 




Fairfax, VA 22033 




(703) 818-1300 



MARRIAGE mi 
FAMILY THERAPY 


Professional Examination Service 
475 Riverside Drive 
New York, NY 10115 
(212) 870-3384 







MEDICINE . 


The Federation of State Medical Boards of the U.S., Inc. 


; '.r’.-fii:™ , , 4 '4;®' 4: ...■ 


400 Fuller Wiser Road, Suite 300 




Euless, TX 76039-3855 


.'v'- ■ 'vr 


(817) 868-4000 


, -...M,.: ' ; 





MENTAL HEALTH 


National Board of Certified Counselors, Inc. 


COUNSELORS 


3-D Terrace Way 


..:v. J.. 


Greensboro, NC 27403 




(910) 547-0607 



MIDWIVERY 


North American Registry of Midwives 




P.O. Box 15 




Linn, WV 26384 



NURSING > 


National Council of State Boards of Nursing 




676 North St. Clair, Suite 550 




Chicago, IL 6061 1-2921 




(312) 787-6555 



NURSING HOME 


National Association of Boards of Examiners for Nursing 


ADMINISTRATORS 


Home Administrators 




808 17th Street, NW, Suite 200 




Washington, DC 20006 


..1. 


(202) 223-9750 
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National Examination Agencies 



mcUPATIONAL i 
I THERAPY J; . 

,::0 , :. - -A, ' ;t ;V 



National Board for Certification in Occupational Therapy 
800 South Frederick Avenue, Suite 200 
Gaithersburg, MD 20877-4150 



PMARMACy ^ 


National Association of Boards of Pharmacy 


' yl>‘ Y" V" 


700 Busse Highway 




Park Ridge, IL 60068 




(847) 698-6227 



PHYSICAL ; 

t X|h|i^pC4. |: 

f if" - i|!' ' 



Professional Examination Service 
475 Riverside Drive 
New York, NY 10115 
(212) 870-3384 



PHYSICIAN ^ 


National Commission on Certification for Physician 


; AS.SISTANI : I 


Assistants 


j-- .'J; ■■„••;•.? ' /'* ', ■i’i:' 


6849 B-2 Peachtree-Dunwoody Road 


: ■ ■■■ ■ !.. . -i j,.: 

■ ■■■■:*!■ ■ ^1?' ■■ ^ 


Atlanta, GA 30328 


i0-: ■■ Jlif ^ • # ■ % ■! ^:hi ■' ■■M-- :!• f ■■■ 


(770) 399-9971 



PSYCHOLOGY 


Professional Examination Service 




473 Riverside Drive 


. I . v' i ■ y 


New York, NY 10115 


j: . .. ^ 


(212) 870-3384 



RESPIRATORY 


National Board of Respiratory Care, Inc. 


■ 'careJ,./' 


8310 Nieman Road 




Lenexa, KS 66214 


I;- rrj; 


(913) 599-4200 



■ ^ SPEECH-’ : 


Educational Testing Services 


. ft ' ”« ^iUU <' *'* <ij$r ' ^ 

language 


Princeton, NJ 


PATHOLOGY k 


(800) 772-9476 



TEACHER 


Educational Testing Services 


ic|jQATipN .i.k 


Praxis Series 


:r:: .jl ‘ ■ . ’V, :|- 


Princeton, NJ 


■!:■■ ■ ';l.l . '.I ' " '■ i 

. .ri ,f i:: ^ ^ ■ m m 


(800) 772-9476 
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National Examination Agencies 



.VETERINARY . 


Professional Examination Services 


MEDICINE 


475 Riverside Drive 


•m- ,|jr .--I f- 


New York, NY 10115 




(212) 870-3384 
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Survey of State Universities and Systems 
Conducted December 1996 to March 1997 



Arizona 

Contact; Arizona Board of Regents 

2020 N. Central Avenue, Suite 230 
Phoenix, AZ 85004-4503 
(602) 229-2500 

The Board of Regents is in the process of developing various measures of performance. 
Performance based budgeting has been looked at by the legislature, but not implemented. After 
reviewing several indicators to possibly use for their measures, the Board decided that licensure 
exam pass rates were not quality measures. 



Arkansas* 

Arkansas is currently computing pass rates on the nursing and national teachers exam. They 
obtain the data from the individual boards responsible for each discipline collected. State Board 
of Nursing and State Department of Education Certification Office, respectively. 



California 

Contact: California Postsecondary Education Commission 

1303 J Street, Suite 500 
Sacramento, CA 95814-2938 
(916) 445-7933 

Currently, the California System does not have any performance based funding initiatives or 
comprehensive accountability model in place for their universities. They are required to publish 
an annual Performance Report that outlines performance indicators used to measure programs 
within the system. Licensure pass rates are not one of the indicators used. The information 
published gives a summary of the system through the following types of information: 
Characteristics of the California population. Fiscal support for the universities. Student 
preparation for college. Student access to college, and Student experiences and outcomes. 



Colorado 



Contact: Colorado Commission of Higher Education 

Colorado History Museum 
1300 Broadway, Second Floor 
Denver, CO 80203 
(303) 866-2723 

This system is currently developing a quality indicator system to be used with performance based 
funding. The committee charged with the responsibility to work on this has recommended that 
the licensure pass rates for teacher education be used as an indicator of those programs in the 
system. The interviewee did caution anyone exploring the possibility of using this type of 
measure not to use it as the only determinant of program quality as they are already experiencing 
problems with this indicator. 



Georgia 

Contact: Board of Regents of the University System of Georgia 

224 Washington Street, SW 
Atlanta, GA 30334 
(404) 656-2202 

The assessment of student learning outcomes is an institutionally defined process. Each 
individual university is responsible for gathering data to assess program quality and student 
learning. At the university level, budgeting is linked to assessment, so indicators must be 
collected. This year, the Office of Fiscal Affairs proposed standard measures for all universities 
to collect, but steered the universities away from gathering information/data on retention, 
enrollment, and licensure pass rates because they “don’t provide you with feedback to improve 
your programs. They might not really indicate student learning or program quality, but issues in 
the economy or society that impact the university.” 



Kentucky* 

SHEEO and the Kentucky Council on Higher Education reports that Kentucky is currently 
collecting pass rates for accountability reporting for fields of dentistry, medicine, law, nursing, 
and teaching. Discussions are ongoing about the possible use of other disciplines. These pass 
rates are used to compare institutions with dental, medical, and law schools based on the number 
of graduates from the previous year who took the respective licensure test for the first time and 
the number who passed. 

Pass rates obtained by the Kentucky Council on Higher Education (CHE) are provided to 
campuses with lists of their graduates of first-professional programs with the following elements: 
major, SSN, last name, first name, took exam (y/n), pass exam (y/n). Institutions are permitted 
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to request amendments to CHE’s analysis if they have documentation of specific exceptions. 



Maryland* 

The public campuses in Maryland are required to report this information as one of the indicators 
in the annual performance accoimtability report submitted to the Commission. Specifically, 
campuses report the percentage of students \vho passed licensure and certification examinations 
in each academic field offered at the institution for \vhich such tests are conducted. The data 
sources used and method of calculation are the responsibility of each campus. 



New York 

Contact: SUNY System Administration 

State Education Department 
Albany, NY 12234 
(518)443-5355 

The Administration is currently studying performance based budgeting, but has not begun any 
formal process for implementation. The Office of Policy Analysis and Planning publishes a 
Performance Indicators Annual Report, but does not use it for funding purposes. It is used more 
as an indication of what/how the system is doing on certain indicators. While the BOR does 
oversee the system, it also has responsibilities for elementary and secondary education, as well. 
In turn, the campuses are more autonomous than is usually the case in a system. Licensure 
information, therefore, is used by individual campuses to assess program quality, teaching, and 
student learning. The BOR does not use that information to assess any campus programs, 
although they do store the information for the imiversities. 



North Carolina 

Contact: University of North Carolina General Administration 

910 Raleigh Road 
Chapel Hill, NC 27515 
(919) 962-1000 

The North Carolina System utilizes licensure pass rates as a component of its University 
Assessment Plan. The system is currently considering the use of licensure pass rates as well as 
other measures in an Incentive Fimding Plan. The UNC-GA collects information on licensure 
pass rates from the individual national organizations of Nursing, Law, and Accoimting. The 
Praxis/NTE scores are collected from the Educational Testing Service. These particular 
academic areas were chosen based upon the essentialness of obtaining a license to work in that 
field. 



Rhode Island* 



Rhode Island does not collect or review the results of professional licensure or certification 
examinations, with the exception of nursing. Pass rates are collected for the nursing examination 
because all three of the public institutions have nursing programs. Since no other examination is 
taken by all students at all institutions, the results are of less interest to Rhode Island Higher 
Education Administration. 



South Carolina 

Contact: South Carolina Commission on Higher Education 

1333 Main Street, Suite 200 
Columbia, SC 29201 
(803) 737-2260 

South Carolina’s university system is currently studying the pros and cons of using specific 
indicators as performance measures in a move to permanently implement performance based 
funding. The legislature, in Spring 1996, passed into law a list of 37 indicators to be used in this 
process. The S.C. Commission now has the responsibility of developing measures, benchmarks, 
weights, etc. for each of these indicators as a part of their effort to implement the proposed 
indicators. One of the indicators reads: “Scores of graduates on post-graduate professional, 
graduate, or employment-related examinations and certification tests. 1) Percentage of total 
students taking certification examinations who pass the examination on the first attempt, and 
2) Percentage of the total students who pass the examination on subsequent attempts.” 



Tennessee 

Contact: Tennessee Board of Regents 

1415 Murfreesboro Road, Suite 350 
Nashville, TN 37217 
(615) 366-4400 

The Tennessee System does utilize licensure pass rates as indicators of program assessment and 
quality. It collects pass rates for the following major areas: Engineering, Allied Health, Law, and 
Medicine. These particular areas were chosen according to the criticality/essentialness of the 
license and whether the licensure agency makes it easy to obtain scores and other information. 
The pass rates are measures used for the performance based funding process set up by the 
legislature. It is just one of 10 variables used to determine allocations of funding for the 
universities in the system. Five percent of funds are determined through this method. The 
System uses the licensure pass scores as a means of making decisions about curricular revisions, 
accreditation, and performance based funding. Tennessee has developed a formula which uses a 
sliding scale based upon number of test taker and pass rates (pass rates with many takers are 
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weighted more heavily; only first-time takers scores are used). The formula is used to generate 
points for performance based fimding. Using licensure pass rates as a measure generates a 
certain number of points to be used toward the performance based funding criteria. 



Texas 

Contact: Texas Higher Education Coordinating Board 

Capitol Station 
Austin, TX 78711 
(512)483-6101 

The Texas System currently uses licensure pass rates as a measure of performance based fimding, 
however there is no link between the measures and money received. The system collects pass 
rate information on the following academic areas: Law, Pharmacy, Nursing, Teacher Education, 
and Engineering. The Legislative Budget Board requests these scores from the national 
agencies, so there is little problem receiving the information on a regular basis. 



West Virginia* 

The State College and University Systems of West Virginia Central Office collect annual data on 
licensure and certification pass rates via paper forms from their 1 6 public institutions. They are 
mandated to collect this data for a legislatively-mandated document entitled, “The Higher 
Education Report Card”. 

The forms used to collect the data contain three elements (# examinees, # passing, and % 
passing) for the period July 1 - June 30. The forms also include a statement explaining that 
“individuals who have taken various licensure exams are not necessarily graduates. They may 
have completed only the course(s) required for licensure in accordance with the licensing 
agency.” 

Forms are collected on 8 licensure exams at the associate’s level (Radiologic Technology, 
Medical Laboratory Technology, Nursing, Medical Records Technology, Nuclear Medicine 
Technology, Land Surveying, Emergency Medical Technology, and Dental Hygiene), 6 licensure 
exams at the bachelor’s level (Medical Technology, Nursing, Dental Hygiene, Social Work, 
Teacher Education, Pharmacy, and Physical Therapy), and 9 licensure exams at the graduate/first 
professional level (Law, Dentistry, Medicine, Teacher Education, National Certified Counselor 
Exam, West Virginia Counselor Exam, Speech Pathology and Audiology, and Family Nurse 
Practitioner). 
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Wisconsin 



Contact: Higher Educational Aids Board 

1 3 1 W. Wilson Street, Room 902 
Madison, WI 53703 
(608) 267-2208 

The Wisconsin System develops an annual system wide accountability report that lists 1 8 
indicators. The system does use licensure pass rates as performance indicators, but only in the 
sense of measuring post-graduate success and only on the Nursing and CPA exams. They chose 
these two areas based on the number of programs within their system. Other programs in the 
system requiring licensure exams were not chosen either because of the low number of programs 
within their system, low number of students in the programs, or because of a problem with 
obtaining the pass rates from licensure agencies. 



* Information was obtained from an informal electronic mail survey conducted by the New York 
State Education Department, Office of Higher Education Research and Information Systems; 
May 1997. 
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Appendix D 



NOTES ON FLORIDA’S EXPERIENCES IN OBTAINING PASS RATE DATA 



Florida’s Department of Business and Professional Regulation (DBPR) includes a Bureau of 
Testing which produces examination statistics for more than 50 professions. Each profession has 
a board that creates their own rules, regulations and methods for administration. Different 
departments within DBPR have different functions; Psychometrics designs the state exams. 
Exams Services administers the exams, and Systems generate statistics from the exam results. 
The reports generated provide information on biographies, education, pass/fail rates, schools, and 
in-state/out-of state status. Copies of the reports are obtained from the Listing and Labels 
department and are available on paper or diskette. 

Unfortunately, the reports currently generated provide very limited feedback when trying to trace 
back to our universities. The biographies such as race, sex, and age are a sum of all the test 
takers and are not presented by university. The State University System (SUS) has been able to 
select First-Time takers who graduated from our universities, but not the date of graduation. 

The SUS collection of information is at the mercy of the sender. During the course of a year, for 
most professions, there are multiple opportunities to sit for an exam. Reports for a particular time 
period can be requested, but if they are not included we must assume that there were no other test 
takers than the reports that we recieved. 

The licensure reports do not tell us whether a person ever manages to pass all the required 
subtests within the prescribed time limit to earn a license. This is important to learn how many 
of our graduates are ever able to practice within their field, if licensing is a requirement. 

Many exams are made up of subtests. If the test taker is required to sit for all the subtests, then 
the report is counting equally. But, if test takers are allowed to sit for part of the exams, then it 
changes the meaning of the pass/fail rates. For instance, if a profession has five subtests and a 
person sits for four and passes only three they are marked as a fail. If another person sits for two 
subtests and passes both, they are considered a pass. 

The first time a candidate sits for the test, they are labeled a first time taker. If it takes three 
sittings before successfully passing all the subtests, they are counted as a fail for First time taker, 
one fail and one pass for repeat takers. 

Some exams require that the person have certain experiences in the field before being eligible to 
sit for the exam. In these cases pass/fail rates should not be used as an instrument to evaluate the 
quality of the educational program. The content of these licensure exams are based on the 
professional activitites which need to be performed correctly by a newly licensed practitioner in 
order to prevent any harm to the public. Emphasis is given to testing the candidate’s clinical or 
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practical knowledge essential to protecting the public. Therefore, licensure exams requiring 
experience in field are not limited to knowledge gained solely from the educational program. 

The greater the time that has passed since completion of the program, the candidate’s 
performance on the exam decreases. Rather than a reflection on the quality of the program it 
indicates that what was learned was not retained over time. On certain exams recent graduates 
who have less practical experience will generally perform less well than their coimterparts with 
more experience. 

The State University System has been fortunate that we are not currently required to report 
licensure for Accoimtability. It is imder consideration for inclusion in Performance Based 
Fimding Reports. However, other levels of post-secondary institutions are required to report 
licensure, such as the commimity college system, the Independent colleges and universities, and 
the vocational institutions. The Board of Regents is often asked to furnish licensure data for the 
legislative staff. The information is also used when conducting program reviews or examining 
programs for various other reasons. 

It has been our experience that getting information from the state agency is very difficult. 
However, in recent years the agency has begim building a computerized data base. At the 
present time, we are working towards acquiring permission to tap into their data base to run 
statistical reports that have meaning. Ideally, we would like to share the data base with the other 
agencies who are required to furnish similar reports. 

Curently, the focus is on the pass rates of first time takers. If we could tap into a data base we 
would be able to expand our knowledge and compare pass rates on sub-tests, to follow 
candidates through the entire process. We could more easily compare our graduates to 
candidates moving into our state who also sit for the exam. We could run statistics on race and 
sex and age. We could compute an average length of time and number of sittings that it takes 
candidates to complete the licensure process. In other words, we would gain a greater 
imderstanding of how licensure affects the process of our graduates ability to work in their field. 
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